Journal of Shanghai Jiao Tong University (Science), 2024 RGB-D Salient Object Detection Lightweight Network Multi-modal Fusion

Asymmetric Dual-Stream Networks for Lightweight RGB-D Salient Object Detection

Wang Yan1   Zheng Wanlu1   Xia Yaozheng1   Wang Shaorong1,2*

1School of Information Science and Technology, Beijing Forestry University
2Engineering Research Center for Forestry-oriented Intelligent Information Processing, National Forestry and Grassland Administration

PDF

TL;DRProposes an asymmetric dual-stream lightweight RGB-D salient object detection network with only 5.1M parameters and 0.77G FLOPs, achieving superior performance across five benchmark datasets.

Abstract

This paper proposes an asymmetric dual-stream lightweight RGB-D salient object detection network. By designing lightweight feature extraction networks of different capacities for each modality, the parameter count is significantly reduced. A Multi-modal Feature Enhancement Fusion (MFEF) module compensates for information loss in the lightweight backbone, while a Global Context Module (GCM) enables dense decoding. Experiments on five benchmark datasets demonstrate that the network outperforms most mainstream models in both quantitative and qualitative metrics, with only 5.1M parameters and 0.77G FLOPs.

Method

RGB-D salient object detection models suffer from increased parameters and computational costs due to multi-modal inputs, making deployment on resource-constrained devices difficult.

Based on the observation that RGB images contain richer semantic information than depth images, design backbone networks of different capacities for each modality (asymmetric design), with the depth backbone having only 55% of the RGB backbone parameters. MFEF and GCM modules compensate for information loss caused by lightweight design.

Asymmetric Backbone

Asymmetric Feature Extraction

Designs MobileNetV3-based backbones of different capacities for RGB and depth streams, reducing depth backbone to 55% of RGB backbone parameters.

MFEF

Multi-modal Feature Enhancement Fusion

Uses inverted residual blocks with attention to select representative RGB features, compensating for lightweight backbone information loss.

GCM

Global Context Dense Decoding

Employs multi-branch multi-scale feature extraction (1x1, 3x3, 5x5, 7x7 dilated convolutions) with dense connections to propagate high-level semantics.

Fig. 1 Overall architecture of lightweight network.

Fig. 1 Overall architecture of lightweight network.

The encoder-decoder network uses MobileNetV3-Large (last three layers removed) for RGB and a lighter network for depth, each producing five-level features. Features are fused via MFEF modules and decoded through GCM dense connections, with predictions at four levels for deep supervision.

Fig. 2 RGB backbone network.

Fig. 2 RGB backbone network.

MFEF first multiplies RGB and depth features element-wise, enhances via IRB, then generates RGB attention through GAP+FC+ReLU+FC+sigmoid. The attention, enhanced features, and raw depth features are combined via IRB to produce fused features that compensate for lightweight backbone limitations.

Fig. 3 Depth backbone network.

Fig. 3 Depth backbone network.

GCM uses dense connections: each decoding sub-module output feeds all subsequent sub-modules. Internally, 1x1 convolution reduces dimensions, then four branches (1x1, 3x3, 5x5, 7x7 dilated convolutions) extract multi-scale features enhanced by channel attention.

Fig. 4 Parameters size of common backbone networks.

Fig. 4 Parameters size of common backbone networks.

Fig. 5 Multimodal feature enhancement fusion module.

Fig. 5 Multimodal feature enhancement fusion module.

Fig. 6 Global context module.

Fig. 6 Global context module.

Results

Parameters
5.1M
Params
59% of MoADNet Ultra-lightweight
FLOPs
0.77G
FLOPs
Lowest Significantly lower
NJU2K
0.038
MAE
Best
LFSD
0.875
S-measure
+1.86% vs MoADNet Leading lightweight
MethodTypeParams(M)FLOPs(G)NJU2K MAENJU2K SDUT SDUT F
UCNetAccurate31160.0430.8970.8630.836
SSFAccurate33460.0430.8990.9150.915
MoADNetLightweight5.01.30.0420.9010.9070.920
MobileSalLightweight6.51.60.0410.9030.8960.912
Ours oursLightweight5.10.770.0380.9040.9210.937

Across five benchmark datasets, the network achieves best or second-best performance on most metrics. On LFSD, compared to MoADNet: MAE reduced 9.37%, S-measure +1.86%, F-measure +2.63%, E-measure +1.77%, while using only 5.1M params and 0.77G FLOPs.

Qualitative Results

Fig. 7 Comparison of salient object detection results for different models.

Fig. 7 Comparison of salient object detection results for different models.

Takeaway

Designing modality-specific backbones of different capacities (asymmetric design) is an effective strategy to reduce parameter redundancy. With well-designed fusion and decoding modules, lightweight networks can match or surpass the performance of complex networks.

Citation

@article{Wang2025,
  title   = {Asymmetric Dual-Stream Networks for Lightweight {RGB-D} Salient Object Detection},
  author  = {Wang, Yan and Zheng, Wanlu and Xia, Yaozheng and Wang, Shaorong},
  journal = {Journal of Shanghai Jiaotong University (Science)},
  year    = {2025},
  doi     = {10.1007/s12204-024-2794-0},
  url     = {https://link.springer.com/article/10.1007/s12204-024-2794-0},
  issn    = {1995-8188},
  abstract= {Integrating image and depth information for RGB-D salient object detection has become a research hotspot in the field of saliency detection. Balancing the efficiency and performance of salient object detection models under resource constraints is a key challenge. To address this, this paper proposes an asymmetric lightweight network suitable for real-time RGB-D salient object detection tasks. The network reduces the number of network parameters by designing different lightweight feature extraction networks for different input modalities. Additionally, a multi-modal feature enhancement fusion module is designed to effectively fuse multi-modal features while compensating for the information loss caused by the lightweight backbone network. Moreover, this paper utilizes a global context module for dense decoding, aggregating local and global information of multi-scale features without significantly increasing computational complexity. The experimental results on five benchmarks show that the proposed lightweight RGB-D salient object detection network not only outperforms most mainstream models quantitatively and qualitatively, but also significantly outperforms other models in terms of efficiency, only with a parameter count of 5.1 million and a computational load of 0.77 gigaflops. This achievement validates the proposed method’s ability to achieve lightweight salient object detection while maintaining high efficiency.}
}